Very large deep learning models trained using gradient descent are remarkably resistant to memorization given their huge capacity, but are at the same time capable of fitting large datasets of pure noise. Here methods are introduced by which models may be trained to memorize datasets that normally are generalized. We find that memorization is difficult relative to generalization, but that adding noise makes memorization easier. Increasing the dataset size exaggerates the characteristics of that dataset: model access to more training samples makes overfitting easier for random data, but somewhat harder for natural images. The bias of deep learning towards generalization is explored theoretically, and we show that generalization results from a model's parameters being attracted to points of maximal stability with respect to that model's inputs during gradient descent.

Deep learning models develop successive representations of their input in sequential layers, the last of which maps the final representation to the output. Here we investigate the informational content of these representations by observing the ability of convolutional image classification models to autoencode the model's input using embeddings existing in various layers. We find that the deeper the layer, the less accurate that layer's representation of the input is before training. Inaccurate representation results from non-uniqueness in which various distinct inputs give approximately the same embedding. Non-unique representation is a consequence of both exact and approximate non-invertibility of transformations present in the forward pass. Learning to classify natural images leads to an increase in representation clarity for early but not late layers, which instead form abstract images. Rather than simply selecting for features present in the input necessary for classification, deep layer representations are found to transform the input so that it matches representations of the training data such that arbitrary inputs are mapped to manifolds learned during training. This work provides support for the theory that the tasks of image recognition and input generation are inseparable even for models trained exclusively to classify.

Supervised deep learning is most commonly applied to difficult problems defined on large and often extensively curated datasets. Here we demonstrate the ability of deep representation learning to address problems of classification and regression from small and poorly formed tabular datasets by encoding input information as abstracted sequences composed of a fixed number of characters per input field. We find that small models have sufficient capacity for approximation of various functions and achieve record classification benchmark accuracy. Such models are shown to form useful embeddings of various input features in their hidden layers, even if the learned task does not explicitly require knowledge of those features. These models are also amenable to input attribution, allowing for an estimation of the importance of each input element to the model output as well as of which inputs features are effectively embedded in the model. We present a proof-of-concept for the application of small language models to mixed tabular data without explicit feature engineering, cleaning, or preprocessing, relying on the model to perform these tasks as part of the representation learning process.

Estimating the probability of failure for complex real-world systems using high-fidelity computational models is often prohibitively expensive, especially when the probability is small. Exploiting low-fidelity models can make this process more feasible, but merging information from multiple low-fidelity and high-fidelity models poses several challenges. This paper presents a robust multi-fidelity surrogate modeling strategy in which the multi-fidelity surrogate is assembled using an active learning strategy using an on-the-fly model adequacy assessment set within a subset simulation framework for efficient reliability analysis. The multi-fidelity surrogate is assembled by first applying a Gaussian process correction to each low-fidelity model and assigning a model probability based on the model's local predictive accuracy and cost. Three strategies are proposed to fuse these individual surrogates into an overall surrogate model based on model averaging and deterministic/stochastic model selection. The strategies also dictate which model evaluations are necessary. No assumptions are made about the relationships between low-fidelity models, while the high-fidelity model is assumed to be the most accurate and most computationally expensive model. Through two analytical and two numerical case studies, including a case study evaluating the failure probability of Tristructural isotropic-coated (TRISO) nuclear fuels, the algorithm is shown to be highly accurate while drastically reducing the number of high-fidelity model calls (and hence computational cost).

机器学习潜力是分子模拟的重要工具，但是由于缺乏高质量数据集来训练它们的发展，它们的开发阻碍了它们。我们描述了Spice数据集，这是一种新的量子化学数据集，用于训练与模拟与蛋白质相互作用的药物样的小分子相关的潜在。它包含超过110万个小分子，二聚体，二肽和溶剂化氨基酸的构象。它包括15个元素，带电和未充电的分子以及广泛的共价和非共价相互作用。它提供了在{\ omega} b97m-d3（bj）/def2-tzVPPD理论水平以及其他有用的数量（例如多极矩和键阶）上计算出的力和能量。我们在其上训练一组机器学习潜力，并证明它们可以在化学空间的广泛区域中实现化学精度。它可以作为创建可转移的，准备使用潜在功能用于分子模拟的宝贵资源。

明显大小的时间变化（称为光曲线）是望远镜在长时间内捕获的感兴趣的观察统计。光曲线提供了空间域意识（SDA）目标（例如对象识别或姿势估计）作为潜在变量推理问题等目标的探索。与较高的精确仪器相比，来自货架上商业架子（COTS）摄像机的地面观测仍然很便宜，但是，有限的传感器可用性与嘈杂的观察结果相结合，可能会产生可能难以建模的gappy时间序列数据。这些外部因素混淆了对光曲线的自动开发，这使光曲线预测和外推成为应用的关键问题。传统上，使用基于扩散或基于示例的方法解决了图像或时间序列的完成问题。最近，由于学习复杂的非线性嵌入方面的经验成功，深度神经网络（DNNS）已成为首选工具。但是，DNN通常需要大量的培训数据，而这些数据不一定在查看单个卫星的光曲线的独特功能时可用。在本文中，我们提出了一种新的方法，可以使用高斯工艺（GPS）预测光曲线的缺失和未来数据点。 GPS是非线性概率模型，可推断后验分布在功能上并自然量化不确定性。但是，GP推理和培训的立方缩放是其在应用中采用的主要障碍。特别是，单个光曲线可以具有数十万个观测值，这远远超出了单个机器上常规GP的实际实现极限。因此，我们采用MUYGP，这是一种可扩展的框架，用于使用最近的邻居稀疏和局部交叉验证的GP模型的超参数估计。 muygps ...

本文通过讨论参加了为期三年的SubT竞赛的六支球队的不同大满贯策略和成果，报道了地下大满贯的现状。特别是，本文有四个主要目标。首先，我们审查团队采用的算法，架构和系统；特别重点是以激光雷达以激光雷达为中心的SLAM解决方案（几乎所有竞争中所有团队的首选方法），异质的多机器人操作（包括空中机器人和地面机器人）和现实世界的地下操作（从存在需要处理严格的计算约束的晦涩之处）。我们不会回避讨论不同SubT SLAM系统背后的肮脏细节，这些系统通常会从技术论文中省略。其次，我们通过强调当前的SLAM系统的可能性以及我们认为与一些良好的系统工程有关的范围来讨论该领域的成熟度。第三，我们概述了我们认为是基本的开放问题，这些问题可能需要进一步的研究才能突破。最后，我们提供了在SubT挑战和相关工作期间生产的开源SLAM实现和数据集的列表，并构成了研究人员和从业人员的有用资源。

目的：目的是将先前验证的深度学习算法应用于新的甲状腺结节超声图像数据集，并将其性能与放射科医生进行比较。方法：先前的研究提出了一种能够检测甲状腺结节，然后使用两个超声图像进行恶性分类的算法。从1278个结节训练了多任务深度卷积神经网络，最初用99个单独的结节进行了测试。结果与放射科医生相当。与培训案例相比，使用来自不同制造商和产品类型的超声计算机成像的378个结节进一步测试了该算法。要求四名经验丰富的放射科医生评估结节，以与深度学习进行比较。结果：用参数，二维估计计算了深度学习算法和四个放射科医生的曲线（AUC）面积。对于深度学习算法，AUC为0.70（95％CI：0.64-0.75）。放射科医生的AUC为0.66（95％CI：0.61-0.71），0.67（95％CI：0.62-0.73），0.68（95％CI：0.63-0.73）和0.66（95％CI：95％CI：0.61-0.71）。结论：在新的测试数据集中，深度学习算法与所有四个放射科医生都达到了类似的性能。

当我们扩大数据集，模型尺寸和培训时间时，深入学习方法的能力中存在越来越多的经验证据。尽管有一些关于这些资源如何调节统计能力的说法，但对它们对模型培训的计算问题的影响知之甚少。这项工作通过学习$ k $ -sparse $ n $ bits的镜头进行了探索，这是一个构成理论计算障碍的规范性问题。在这种情况下，我们发现神经网络在扩大数据集大小和运行时间时会表现出令人惊讶的相变。特别是，我们从经验上证明，通过标准培训，各种体系结构以$ n^{o（k）} $示例学习稀疏的平等，而损失（和错误）曲线在$ n^{o（k）}后突然下降。 $迭代。这些积极的结果几乎匹配已知的SQ下限，即使没有明确的稀疏性先验。我们通过理论分析阐明了这些现象的机制：我们发现性能的相变不到SGD“在黑暗中绊倒”，直到它找到了隐藏的特征集（自然算法也以$ n^中的方式运行{o（k）} $ time）;取而代之的是，我们表明SGD逐渐扩大了人口梯度的傅立叶差距。

Transfer learning increasingly becomes an important tool in handling data scarcity often encountered in machine learning. In the application of high-throughput thickness as a downstream process of the high-throughput optimization of optoelectronic thin films with autonomous workflows, data scarcity occurs especially for new materials. To achieve high-throughput thickness characterization, we propose a machine learning model called thicknessML that predicts thickness from UV-Vis spectrophotometry input and an overarching transfer learning workflow. We demonstrate the transfer learning workflow from generic source domain of generic band-gapped materials to specific target domain of perovskite materials, where the target domain data only come from limited number (18) of refractive indices from literature. The target domain can be easily extended to other material classes with a few literature data. Defining thickness prediction accuracy to be within-10% deviation, thicknessML achieves 92.2% (with a deviation of 3.6%) accuracy with transfer learning compared to 81.8% (with a deviation of 3.6%) 11.7% without (lower mean and larger standard deviation). Experimental validation on six deposited perovskite films also corroborates the efficacy of the proposed workflow by yielding a 10.5% mean absolute percentage error (MAPE).